Skip to content

0.6.0: LocalVerifier (score locally with your fine-tuned model) + hosted verify() sends similarity/cached_query - #7

Merged
imxinchengyou merged 1 commit into
mainfrom
feat/local-verifier
Oct 31, 2026
Merged

imxinchengyou merged 1 commit into
mainfrom
feat/local-verifier

Conversation

@imxinchengyou

Copy link
Copy Markdown
Owner

Why

This is the client side of local scoring for Managed plans; the service side is cacheverifier-service #128 and #129. A tenant scores gray-zone hits on its own machines with the model the service fine-tuned for it. Fine-tuning, drift monitoring, audits and certification keep running on the service.

Changes

  • New cacheverifier.local.LocalVerifier, also exported from the package root:
    • Model loading: downloads the live model from GET /v1/models/active and /v1/models/{id}/files/{name}. An interrupted download resumes with a Range request, and every file is checked against its sha256. It then loads the model with sentence-transformers (new local extra, the same stack as healthcheck).
    • Reference check: before deciding anything it reproduces the manifest's reference verdicts. Each local score must be within 1e-3 of the service's, and the decision rules must give the service's verdict on the service's own score. A model that fails is not used (ModelCheckFailed).
    • Model refresh: a background thread checks for a newer live version every 10 minutes and switches to it once it has passed the check. It keeps two versions on disk.
    • Decisions: same arguments and VerifyResult as the hosted verify(). It decides with the threshold, joint and gap rules (floor included), with the service's precedence.
    • Audits: while audit_rate > 0 it draws audits locally and queues each decision: score, rule, verdicts, audit probability, and an HMAC of the pair under the tenant's pair_key, never the texts. The queue is uploaded to POST /v1/verify/decisions every 30 seconds, or sooner when 1,000 have built up, and holds at most 100k decisions during an outage. feedback(..., result=r) uploads a still-pending decision first, then links the feedback with its verification_id.
    • Fallback: fallback="hosted" (default) sends calls to the hosted API until a model is ready; "reject" returns a degraded rejection. wait=True blocks until the first model is ready.
    • ACI: not supported locally. It logs a warning when the account has ACI on.
  • New cacheverifier._inference: a pure-Python port of the service's inference path, namely compose_query_input with the cached_query diff, smart_truncate_pair, JointRule and decide. The healthcheck now imports its truncation from here.
    • This fixes a drift: the healthcheck's copy predated the service's verifier-core 0.7.0 change, where a long query next to a short answer was cut to half the budget.
  • Hosted client parity with the API:
    • verify() takes similarity_score, cached_query and runner_up_similarity_score. Before this, the client could not send them, so a fine-tuned model's joint rules never applied to SDK calls.
    • feedback() takes cached_query and verification_id.
    • VerifyResult gains decision_rule, audit and verification_id.
  • Version: 0.6.0. README and changelog are updated.

Testing

  • tests/test_local.py: 9 offline tests, using a mock service and a word-overlap model.
    • download, check and local decisions, with no hosted calls;
    • resuming a download with Range;
    • a model whose references don't match is not used, and calls fall back;
    • a 402 keeps the client on the fallback;
    • audits upload without texts, with the right HMAC, and feedback links;
    • nothing is uploaded with auditing off;
    • a newer version replaces the old one, and old downloads are pruned;
    • decision precedence and the floor;
    • the new truncation.
  • Parity on a real fine-tuned NLI checkpoint: the service's verifier-core and app.local_bundle.REFERENCE_PAIRS against this client:
    • with a joint rule and a gap rule, both with floors, and with and without cached_query, the check passes, the scores differ by at most 1.4e-06, and all 20 verdicts are equal, covering all three rules;
    • truncation matches on 300 random long pairs;
    • the cached_query composition matches.
  • Suite: 55 passed, 1 skipped. ruff and mypy are clean. On Python 3.9 (Docker), test_local and test_client pass.

Release

Don't release until the service side (#128, #129) is deployed. It is now: #128 is live and #129 is waiting to merge.

…y() sends similarity and cached_query; healthcheck truncates like the service
@imxinchengyou
imxinchengyou merged commit dc98921 into main Oct 31, 2026
7 checks passed
@imxinchengyou
imxinchengyou deleted the feat/local-verifier branch October 31, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant